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Abstract 



An information reproducing apparatus comprising a program storage device (11) for storing a program, a 
program information outputting device (13) for generating and outputting program information on the basis of 
information which is recorded together with the program in the program storage device (11), and a program- 
associated information reproducing device (15) for selecting suitable program-associated information according 
to the program information, and reproducing a suitable portion of the program-associated information according 
to the program information. According to the information reproducing apparatus of the present invention, a TV 
program broadcasted by a broadcast station or the like and program-associated information which is generated 
in association with the program by the broadcast station itself or a third party other than the broadcast station can 
be received by viewership while the program-associated information stored in PC or the like is reproduced in 



synchronism with the reproduction of the TV program recorded in VCR or the like. 
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Information reproducing apparatus 


* 



(57) An information reproducing apparatus compris- 
ing a program storage device (11 ) for storing a program, 
a program information outputting device (13) for gener- 
ating and outputting program information on the basis 
of information which is recorded together with the pro- 
gram in the program storage device (11), and a pro- 
gram-associated information reproducing device (15) 
for selecting suitable program-associated information 
according to the program information, and reproducing 
a suitable portion of the program-associated information 



according to the program information. According to the 
information reproducing apparatus of the present inven- 
tion, a TV program broadcasted by a broadcast station 
or the like and program-associated information which is 
generated in association with the program by the broad- 
cast station itself or a third party other than the broadcast 
station can be received by viewership while the pro- 
gram-associated information stored in PC or the like is 
reproduced in synchronism with the reproduction of the 
TV program recorded in VCR or the like. 
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Description 

BACKGROUND OF THE INVENTION 

Field of the Invention: 

The present invention relates to an apparatus and 
a method for reproducing and displaying program-asso- 
ciated information in synchronism with a TV (television) 
program which is reproduced by a program storing de- 
vice such as VCR in a system which comprises a TV 
program storing device such as VCR and/or a data in- 
formation displaying device such as PC (personal com- 
puter). 

Description of the Related Art: 

As one of conventional methods for reproducing the 
program-associated information in synchronism with a 
TV program, there was a method which comprises em- 
bedding data of program -associated HTML file in verti- 
cal blanking interval of a TV signal in a broadcast station, 
transmitting the TV signal from a broadcast station, re- 
ceiving the TV signal by a PC-based receiver, separat- 
ing the data of program-associated HTML, and display- 
ing the data of HTML file in synchronism with a TV pro- 
gram on a display of PC. 

However, in the conventional method as described 
above, if there is an intervention of VCR between a 
broadcast station and PC, it is impossible to display the 
data of HTML file with synchronism with a TV program 
because of the loss of the data of HTML file in VCR. A 
"part of the^ata^f'HTMlf file irvvertical blanking interval" 
is lost at switching position between a field and another. 

Further, the above problem makes it difficult for a 
third party such as IP (information provider) to provide 
the program-associated information. In general, IP can- 
not know the contents of a TV program in advance, and 
thus IP provides program-associated information sub- 
sequently to the broadcast of the TV program. In order 
to synchronously reproduce the program-associated in- 
formation and the TV program in a home for example, 
the TV broadcasting program must be temporarily re- 
corded in a program storage device such as a VCR or 
the like. However, there has not been hitherto provided 
any method for synchronously reproducing TV pro- 
grams stored in the VCR or the like and the program- 
associated information supplied from IP. 

SUMMARY OF THE INVENTION 

The object of the present invention is to provide an 
apparatus and a method for reproducing and displaying 
the prog ram -associated information in synchronism 
with a TV program which is reproduced by a program 
storing device such as VCR in a system which compris- 
es a TV program storing device such as VCR and/or a 
data information displaying device such as PC. 



According to the present invention, there is provid- 
ed an information reproducing apparatus which com- 
prises a program storage device for storing programs, 
a program information output device for generating and 

s outputting program information containing program ID 
information, in-program time information, etc. on the ba- 
sis of information which is stored together with the pro- 
grams in the program storage device, and a program- 
associated information reproducing device for selecting 

10 suitable program-associated information according to 
the program ID information and reproducing a suitable 
portion of the program-associated information accord- 
ing to the in-program time information. 

15 BRIEF DESCRIPTION OF THE DRAWINGS 

Fig. 1 is a diagram showing first to fifth embodi- 
ments of the information reproducing apparatus ac- 
cording to the present invention; 
20 Fig. 2 is a diagram showing a second embodiment 
of the information reproducing apparatus of the 
present invention; 

Fig. 3 is a diagram showing a third embodiment of 
the information reproducing apparatus according to 
25 the present invention; 

Fig. 4 is a diagram showing a fourth embodiment of 
the information reproducing apparatus according to 
the present invention; 

Fig. 5 shows an example of program information of 

so the present invention; 

Fig. 6 is a diagram showing the inner construction 
of a PC, software on the PC and the transmission 
— — of each kind of information therebetween when 
each of the first to fifth embodiments is implemented 

35 by a PC; 

Fig. 7 is a flowchart showing an algorithm for gen- 
erating program ID information and in-program time 
information by using audio signals as a cue in the 
present invention; 

40 Fig. 8 shows an example of a partial audio pattern; 
Fig. 9 shows a first example of an overall audio pat- 
tern; 

Fig. 1 0 shows a second example of the overall au- 
dio pattern; 

45 Fig. 11 shows a third example of the overall audio 
pattern; 

Fig. 12 is a flowchart when the block 102 of Fig. 7 
is expanded to processing using video and audio 
data; 

50 Fig. 1 3 is a diagram showing a display frame which 
is displayed by the information reproducing appa- 
ratus of the presents invention; and 
Fig. 14 is a diagram showing another display frame 
which is displayed by the information reproducing 

55 apparatus of the presents invention. 
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DESCRIPTION OF PREFERRED EMBODIMENTS 

Preferred embodiments according to the present in- 
vention will be described hereunder with reference to 
the accompanying drawings. 

First, the preferred embodiments of the present in- 
vention will be briefly explained with reference to Fig. 1 . 

Reference numeral 11 represents a program stor- 
age device such as VCR or the like. For example, a vid- 
eo portion and an audio portion of broadcasted TV sig- 
nals are recorded and stored in program storage device 
11. Further Reference numeral 12 represents a trans- 
mission path for transmitting an audio signal, a video 
signal or a control signal which is output from the pro- 
gram storage device 11 . 

Reference numeral 1 3 represents the program in- 
formation outputting device for generating program in- 
formation such as program ID information of a program 
being currently reproduced in the program storage de- 
vice 11 and in-program time information representing 
which portion in the program is being reproduced on the 
basis of the signal from the transmission path 1 2, and 
for outputting the program information to program-asso- 
ciated information reproducing device 15. Program in- 
formation outputting device 13 may be constructed by 
a software which operates on a PC, and it may be con- 
structed integrally with program storage device 11. 

Reference numeral 14 represents a transmission 
path for transmitting the program information such as 
the program ID information, the in-program time infor- 
mation, etc. which are generated by the program infor- 
mation outputting device 13. 

- Reference numeral T7-represents a storage device, 
and serves to store program-associated information 
supplied from the external. Storage device 17 may be 
constructed by a PC data storage device such as a hard 
disc or the like. Storage device 17 may be separated 
from the other units represented by the block diagrams 
of Fig. 1 and constructed by a server device at a remote 
place. In this case, transmission path 16 may be con- 
structed by a telephone line or the like. 

Link 16 represents a transmission path for transmit- 
ting to the program-associated information reproducing 
apparatus the program-associated information stored in 
storage device 17. 

Reference numeral 15 represents a program-asso- 
ciated information reproducing device for reproducing 
the program-associated information stored in storage 
device 17. On the basis of the program ID information 
on the transmission path 14, program-associated infor- 
mation reproducing device 15 first selects the corre- 
sponding program-associated information from plural 
pieces of program-associated information stored in stor- 
age device 17, and determines on the basis of the in- 
program time information on the transmission path 14 
which portion in the program-associated information is 
reproduced. The program-associated information re- 
producing device 15 may be constructed by a software 



which operates on the PC, and the program information 
outputting device 1 3 and the program-associated infor- 
mation reproducing device 15 may be constructed to op- 
erate on the same PC. 

5 Link 1 8 represents a transmission path for transmit- 
ting video and audio signals which are obtained by re- 
producing program-associated information in program- 
associated information reproducing device 15. 

Link 1 9 represents a transmission path fortransmit- 

10 ting video and audio signals of programs which are out- 
put from the program storage device 11 

Reference numeral 20 represents an output device 
for outputting the video and audio data of the transmis- 
sion paths 18 and 19 at the same time. For example, 

is the output device 20 may be constructed by arranging 
two CRT monitors, or by one TV monitor having a func- 
tion of receiving signals from the two paths 18, 19 and 
integrally displaying these signals on one frame. Fur- 
ther, the output device 20 may be constructed as an out- 

20 put device serving as a part of the PC. 

There is a method of using auxiliary information 
when program information outputting device 1 3 gener- 
ates program information such as program ID informa- 
tion, in-program time information, etc. The following five 

25 modes on the treatment of the auxiliary information may 
be used. 

In a first mode, no auxiliary information is used. This 
mode may be used when the transmission contents of 
transmission path 12 is identical to that of transmission 

30 path 1 4, or the information of transmission path 14 can 
be generated from the information of transmission path 
1 2 without using any auxiliary information. For example, 
in the case that program ID information and in-program 
time information are beforehand encoded and embed - 

35 ded in a vertical blanking interval of a TV broadcasting 
program transmitted from a TV broadcast station in such 
a format that these information can be directly recorded 
in a VCR, the program information outputting device 1 3 
can generate the information of the transmission path 

40 1 4 with no assistance of the auxiliary information by in- 
putting video signals from the transmission path 1 2 if the 
program information outputting device 1 3 has a function 
of decoding the above information (the program infor- 
mation and the time information). In this case, program 

45 id information and in-program time information may be 
embedded in a vertical blanking interval except switch- 
ing position because the amount of information of them 
is far less than that of HTML file. This mode is shown in 
fig. 1. 

50 The construction of program storage device 11 and 
the program information outputting device 1 3 shown in 
Fig. 1 is modified as shown in Fig. 2 to form a second 
mode. In this mode, auxiliary information generated at 
the program storing time is also stored in program stor- 
es age device 11, and the auxiliary information is also re- 
produced at the program reproduction time. On the ba- 
sis of the reproduced auxiliary information, program in- 
formation outputting device 1 3 calculates program ID in- 
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formation, in-program time information, etc. 

In Fig. 2, reference numeral 21 represents a time 
counter for counting current time. Link 22 represents a 
transmission path for transmitting time information out- 
put from time counter 21 to program information output- 
ting device 1 3. 

Reference numeral 23 represents a channel detect- 
ing device for detecting a channel which is selected dur- 
ing the recording operation of program storage device 
1 1 . The information on the channel can be obtained by 
checking the status of the VCR, monitoring record set- 
ting information to the VCR transmitted from a remote 
controller or the like, or by user's direct specification in- 
dependently of the setting to the VCR. 

Link 24 represents a transmission path for transmit- 
ting the channel information from channel detecting de- 
vice 23 to program information outputting device 13. 

Time counter 21 and channel detecting device 23 
may be constructed integrally with program storage de- 
vice 11 . Time counter 21 may be replaced by a function 
which is supplied by the PC. Channel detecting device 
23 may be constructed by a software on the PC. 

Link 25 represents a transmission path for transmit- 
ting auxiliary information as follows. The program infor- 
mation output device 13 converts program information 
such as program ID information, in-program time infor- 
mation, etc. to auxiliary information which can be record- 
ed together with a program during a recording operation 
in program storage device 11, and then transmits the 
auxiliary information thus generated through transmis- 
sion path 25 to program storage device 11. 

In a program reproducing operation, program stor- 
age device 11 also reproduces the auxiliary information 
which has been converted in such a format that it can 
be recorded in program storing device 11 in the record- 
ing operation, and outputs the auxiliary information thus 
reproduced through transmission path 1 2 to program in- 
formation outputting device 1 3. For example, the auxil- 
iary information is embedded in a vertical blanking in- 
terval so that the auxiliary information can be recorded 
in program storage device 11 . 

Program storage device 11 records the auxiliary in- 
formation received from transmission path 25 simulta- 
neously with the recording operation of the program. 
Transmission paths 12 and 25 may be constructed by a 
physically single bi-directional transmission path. 

Construction of the program storage device 11 and 
program information outputting device 13 shown in Fig. 
1 may be modified as shown in Fig. 3 to form a third 
mode. 

In Fig. 3, reference numeral 26 represents a storage 
device for auxiliary information generated by program 
information outputting device 1 3 in the program storing 
operation. 

Link 27 represents a transmission path which links 
program information outputting device 1 3 and storage 
device 26. Storage device 26 may be merged to storage 
device 17. Reference numerals 21 to 24 represent the 



same units as shown in Fig. 2. 

At the same time when program storage device 11 
stores a program, program information outputting de- 
vice 1 3 obtains the contents of the program. Program 
s information outputting device 1 3 generates a overall cue 
signal for a overall program and segment cues for seg- 
ments of a program divided by time on the basis of 
closed caption or video and audio data of a program, 
associates program ID information to the overall cue 
io and in-program time information to segment cues. The 
pair of an overall cue and program ID information and 
pairs of segment cue and in-program time information 
constitutes auxiliary information. Thereafter, Program 
information outputting device 1 3 stores auxiliary infor- 
ms mation in storage device 26. In the reproducing opera- 
tion, program information outputting device 13 com- 
pares the information such as the closed caption, etc. 
obtained from program storage device 11 with the aux- 
iliary information read out from storage device 26 by a 
20 pattern matching method or the like to judge which por- 
tion in the auxiliary information corresponds to the infor- 
mation, and generates and outputs the corresponding 
program information such as the program ID informa- 
tion, the in-program time information, etc.. 
25 The construction of first program storage device 11 
and program information outputting device 1 3 shown in 
Fig. 1 is modified as shown in Fig. 4 to form a fourth 
mode. In Fig. 4, storage device 26 and transmission 
path 27 are the same as those of Fig. 3. 
so In this embodiment, the auxiliary information gen- 
erating processing is not performed in the program in- 
formation outputting device 13 in the program storing 
operation, and the auxiliary information is generated at 
the external. The auxiliary information is supplied to 
35 storage device 26 from the external at a timing which is 
independent of the program storing operation, or pro- 
gram information outputting device 1 3 obtains the aux- 
iliary information from the external and then stores it into 
storage device 26. In the reproducing operation, the pro- 
40 gram ID information and the in-program time information 
are generated in the same procedure as described in 
the third mode. In this mode, the processing which is 
required in the recording operation in the third mode can 
be omitted because the auxiliary information is gener- 
is ated in the external. 

In a fifth mode, the construction is the same as that 
shown in Fig. 1 but the auxiliary information is used. 

The auxiliary information generating processing is 
not performed in the program information outputting de- 
so vice 1 3 in the program storing operation as in the case 
of the fourth mode. In the reproducing operation, pro- 
gram information outputting device 1 3 connects to an 
external server by using transmission means such as a 
telephone line or the like to obtain auxiliary information 
ss and generates program information such as program ID 
information and in-program time information referring to 
auxiliary information thus obtained. On the contrary to 
the fourth mode requiring the process of obtaining aux- 
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iliary information before the reproducing operation, the 
fifth mode does not need this process until the repro- 
ducing operation. In addition, on the contrary to the 
fourth mode in which the communication for download- 
ing the auxiliary information may be performed at any 
time from the time when the auxiliary information is pre- 
pared at the external and until the reproducing operation 
is performed in the fourth mode, according to the fifth 
mode, the communication must be performed during the 
reproducing. 

Fig. 5 shows an example of prog ram -associated in- 
formation which is read by program-associated informa- 
tion reproducing device 15. 

In Fig. 5, data 81 represents header information cor- 
responding to program ID information. If there is a code 
system which can specify a program uniquely, the code 
of the system can be used as program ID information. 
Further, information set consisting of program transmis- 
sion media, channel, storing start time, etc. which can 
specify the program uniquely may be used as program 
ID information. 

Data 82 describes the total number of sets of in- 
program time information and reproduction information. 
Data 83 represents in-program time information. 

Data 84 represents reproduction information which 
specifies the content to be reproduced at the time when 
the in-program time information 83 is specified. This in- 
formation contains texts which are described in various 
formats, figure drawing commands, video file names, 
etc. The reproduction information 84 may be described 
by means of hypertext style data (for example, HTML) 
in which another information is supplied by the operation 
of inputting~by a~ user, for example; clicking a mouse on 
a specific item. 

Here, an example of the method of generating pro- 
gram ID information 81 and in-program time information 
83 with the assistance of the audio signal as a cue. 

In this embodiment, program information outputting 
device 13 and program-associated information repro- 
ducing device 15 shown in Fig. 1 are assumed to be 
implemented by different software on the same PC. The 
relationship between the constituent elements of the 
PC, the software on the PC and various information 
transmitted therebetween when program ID information 
81 and in-program time information 83 are generated 
with the assistance of the audio signal as a cue will be 
described with reference to Fig. 6. 

Reference numeral 151 represents a audio input 
device in the PC, and audio input device 151 comprises 
a audio A/D converter and a data buffer. Audio input de- 
vice 151 is connected to program storage device 11 
through the link 12. 

Reference numeral 152 represents an input/output 
device having a network function, a modem function, 
etc. in the PC to communicate to a remote place. Ref- 
erence numeral 153 represents a storage device such 
as a hard disc or the like which is mounted inside the 
PC or provided outside of the PC, and it is designed to 



unify storage device 17 shown in Fig. 1 and storage de- 
vice 26 shown in Fig. 3. 

Reference numeral 154 represents an output de- 
vice in the PC, and outputs video and audio signals to 
s the external. Output device 1 54 is connected to output 
device 20 of Fig. 1 through the link 18. 

Reference numeral 155 and reference numeral 156 
represent programs which operate on the PC. Informa- 
tion generating program 1 55 is the software correspond- 
io ing to the function of program information outputting de- 
vice 13 of Fig. 1, and the information reproducing pro- 
gram 156 is the software corresponding to the function 
of the program-associated information reproducing de- 
vice 15. 

Reference numeral 161 represents transmission of 
audio data. Link 162 represents transmission of auxilia- 
ry information when the auxiliary information generated 
by information generating program 155 is stored into 
storage device 1 53, and the auxiliary information stored 
in storage device 1 53 is referred to by information gen- 
erating program 155. 

Link 163 represents program ID information 81 and 
in-program time information 83 which are output to in- 
formation reproducing program 156 by information gen- 
erating program 155. Link 164 represents transmission 
of program-associated information which is obtained 
from storage device 153 by the information reproducing 
program 1 56. Link 1 65 represents transmission of texts, 
figures, image drawing command, etc. which are issued 
on the basis of the program-associated information by 
information reproducing program 156. Link 166 repre- 
sents transmission of program-associated information 
and the auxiliary information which are obtained through 
the input/output device 152 and stored in storage device 
153. *s 

Fig. 7 shows an algorithm of generating the pro- 
gram ID information and the in-program time information 
by information generating program 155 shown in Fig. 6 
while VCR is under reproducing operation. 

Here, the cue information on the overall program 
contained in the auxiliary information is hereinafter re- 
ferred to as "overall audio pattern", and the cue infor- 
mation corresponding to a partial time width which is ob- 
tained from VCR at the reproduction time of the VCR is 
hereinafter referred to as "partial audio pattern". 

In Fig. 7, in step 101, audio data having a certain 
time width are newly obtained from audio input device 
151 which corresponds to transmission path 12. This 
audio data corresponds to "partial audio pattern". In step 
102, the pattern matching judgment of the partial audio 
pattern obtained in step 1 01 is performed for all the con- 
tents of plural overall audio patterns serving as synchro- 
nism judgment candidates which has been prepared by 
information generating program 155 at the program re- 
cording time or which are beforehand inputted from the 
external and stored in storage device 153. If no match- 
ing is established for all overall audio patterns, i.e. the 
judgment in step 102 being No, the process returns to 
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step 101. If the matched portion is detected, i.e. the judg- 
ment in step 102 being YES, the process proceeds to 
step 103. Here, in order to enhance the matching effi- 
ciency, the following method may be used. That is, when 
plural matching candidates are temporarily obtained, 5 
but it is not possible to narrow down to one candidate, 
the judgment in step 102 is temporarily set to No, and 
the process returns to step 101 . When the process goes 
to step 1 02 again, a pattern subsequent to the matching 
candidates which are previously obtained from the over- 
all pattern is preferentially subjected to the matching 
judgment. This method may be applied in step 108. 

In step 103, the program ID information 81 such as 
the program name, etc. on the program corresponding 
to a currently inputted audio and the in-program time in- 
formation 83 on the time corresponding to the content 
being reproduced in the program are output to the ex- 
ternal on the basis of the information obtained in step 
102. 

Steps 101 to 103 may be omitted when the program 
ID information, etc. are supplied from the external. In 
this case, the overall audio pattern which is subjected 
to the pattern matching in steps 105 and 108 is deter- 
mined uniquely on the basis of the program ID informa- 
tion 81 which is supplied from the external. 

Step 104 is identical to step 101. In step 105, it is 
judged whether a pattern subsequent to the latest 
matched portion of the overall audio pattern for which 
the matching was accomplished in step 102, 105 or 108 
matches with a partial audio pattern which is newly ob- 
tained at this time, i.e. whether the synchronism being 
maintained. If the matching is detected, i.e. the judg- 
ment in step 105 being YES, the process proceeds to 
step 106. If not so, i.e. the judgment in step 105 being 
NO, the process proceeds to step 107. In step 106, the 
in-program time information 83 which corresponds to 
the currently reproduced content in the program settled 
in step 102 is output to the external on the basis of the 
information obtained in step 105 or 108. In step 107, it 
is judged whether the continuous time of non-detection 
in step 105 exceeds a certain time or the frequency of 
non -detection in step 105 exceeds a certain value. If the 
judgment in step 107 is YES, the process proceeds to 
step 108. If not so, i.e. the judgment in step 107 being 
NO, the process proceeds to step 1 04. Step 1 07 is in- 
troduced to prevent the process from easily shifting to 
step 108 which has a relatively high processing load 
when the judgment of step 105 is not correctly made at 
a time due to a sudden noise or the like, whereby the 
processing efficiently can be totally enhanced, and the 
average time required until the matching is established 
can be shortened. 

Step 108 performs the pattern matching between 
the current partial audio pattern and the overall audio 
pattern of the program obtained in step 1 02. In step 1 09, 
it is judged whether the continuous time of non-detection 
in step 108 exceeds a certain time or the frequency of 
non-detection in step 108 exceeds a certain value. If the 



judgment in step 109 is YES, the process proceeds to 
step 120. If not so, i.e. the judgment in step 107 being 
NO, the process proceeds to step 1 02. The operation of 
step 109 is identical to that of step 107. The operation 
of step 120 is identical to that of step 101 . 

In the mode shown in Fig. 7, once the matching is 
established (the synchronism is established), the judg- 
ment of the synchronism maintenance is first performed. 
If it is judged that the synchronism is not maintained, the 
matching judgment in the same program is performed. 
One of the reasons why synchronism breaks is that a 
VCR goes under special reproduction mode such as 
stop mode, slow mode, and picture search mode. When 
the VCR returned to normal reproduction mode again, 
the reproduction may restarts from a point which is far 
away from the point to which the previous synchronism 
had been continued, and thus the every portions of the 
same program are targeted for the matching. However, 
in a medium having a sequential access such as a VCR 
or the like, there is a certain correlation between the time 
period for which the synchronous state breaks due to 
the special reproduction and the time difference be- 
tween the point to which the previous reproduction had 
been performed and the point from which the reproduc- 
tion restarts, so that the matching efficiency can be en- 
hanced by utilizing this correlation. For example, in such 
a condition that only one second has just elapsed from 
the break of the synchronism, it is most probable that 
the reproduction at a point near to the final point to which 
the synchronism had been continued restarts. There- 
fore, the matching judgement is performed in the corre- 
sponding portion of the same program. 

When the matching is not established in the same 
program, the matching judgment is performed on all the 
programs. The situation by which the program is 
changed to another is that the reproduction of VCR or 
the like extends to a next program which is continuously 
recorded subsequent to the present program, or that a 
tape is exchanged by another. In this embodiment, the 
matching judgment processing in the same program is 
more preferentially performed than the matching judg- 
ment processing on all the programs on the assumption 
that the change of the program as described above is 
more rarely performed than the shift in the same pro- 
gram due to the special reproduction. 

Next, an embodiment of the process of obtaining 
program ID information 81 and in-program time informa- 
tion 83 by the pattern matching in step 102 of fig. 7 will 
be explained. 

Fig. 8 shows a diagrammatic picture of an example 
of the PCM data of the partial audio pattern obtained in 
the reproduction operation. The ordinate represents a 
sound level (volume) axis, and the abscissa represents 
a time axis. Here, it is assumed that there are programs 
a, b and c as program candidates to be reproduced, and 
the overall audio patterns of all these programs have 
been already stored in the storage device 26. Figs. 9, 
1 0 and 1 1 schematically show the PCM data of the over- 
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all audio patterns of the three kinds of programs a, b and 
c. In auxiliary information containing the overall audio 
pattern, program ID information 81 is set as header in- 
formation. When a code system which can specify any 
program uniquely exists, the code of the system may be 
used as the program ID information. Besides, an infor- 
mation set which consists of program transmission me- 
dia, channel, storing start time, etc. and can specify any 
program uniquely may be used as the program ID infor- 
mation. 

By performing the pattern matching judgment on 
the overall data contained in Figs. 9, 10 and 11 1 it can 
be found out which portion in the three overall audio pat- 
terns is most matched with the partial audio pattern of 
Fig. 8. In this example, the portion surrounded by a 
square of Fig. 10 is coincident with the partial audio pat- 
tern of Fig. 8. In this case, program ID information 81 of 
the program b can be obtained from the header infor- 
mation of the overall audio pattern file of program b. 
Since the abscissa of Fig. 11 shows the time axis, in- 
prog ram information 83 of the portion surrounded by the 
square can be obtained. 

In the processing which is performed in step 108 of 
Fig. 7, the target file to be subjected to the pattern 
matching is limited to one file. In this example, it is the 
file of program b in Fig. 10. 

In order to enhance the matching precision while 
increasing the matching speed, it is necessary to set the 
sampling frequency and the quantizing bit number of 
PCM data to the optimum values. Further, in order to 
enhance the matching precision, mere PCM data on the 
time axis are not used as the data format for comparison 
between data, but plural power data calculated on plural 
frequency components by using the band pass filters 
which performs FFT or the like are used. 

Even for data which are originally generated from 
the same program, there is a possibility that the volume 
level and the steady noise of a audio signal of the overall 
audio pattern varies from those of the partial audio pat- 
tern due to the fluctuation of environments when data 
are generated or the like. In this case, it is preferable 
that the difference in level between both the data is cor- 
rected or the effect of the steady noise is subtracted 
when both of the data are used for matching judgement. 

According to the method as mentioned above, the 
synchronization breaks during the special reproduction 
such as the picture search, and the synchronism is es- 
tablished again at the time point when the normal repro- 
duction restarts. But it is possible to maintain synchro- 
nization during the special reproduction as long as an 
audio signal is supplied. In this case, an audio signal of 
which time axis is compressed/expanded or reversed 
must be converted to that of the normal reproduction to 
be used for matching judgement in step 108. 

In the above embodiment, the audio data are tar- 
geted. If the audio data pickup processing and the 
matching judgment processing of the audio data in the 
embodiment shown in Fig. 7 is replaced by video data 



pickup processing and matching judgment processing 
of video data, the pattern matching on video data can 
be performed. The method of performing the pattern 
matching of the video data is particularly effectively it it 
s is used for the programs in which variation of audio pat- 
terns is little, similar patterns appear frequently, or a 
non-sound portion continues. 

In the case of the matching operation based on pic- 
tures, average brightness of continuous fields of pic- 
10 tures is used for matching judgement instead of audio 
data. Further, in order to enhance the matching preci- 
sion, plural data set which is composed of average lu- 
minance levels or average levels of RGB component for 
divided areas of a field may be used. 
is Further, the matching of the audio data and the 
matching of the video data may be used in combination 
to enhance the efficiency of the matching. If the audio 
data pickup processing and the matching judgment 
processing based on audio data in the embodiment 
20 shown in Fig. 7 are respectively replaced by the audio 
and video data pickup processing and a pair of the 
matching judgment processing based on audio data and 
the matching judgment processing based on picture da- 
ta, the embodiment shown in Fig. 7 can be modified to 
2S an embodiment using both audio and picture data. 

In this modification, the matching of both of the au- 
dio data and picture data is performed at all the time in 
the matching judgment operation, and if the matching 
judgment is made on any of audio data and picture data, 
30 the matching is regarded as being established. If the 
matching judgment is made on both of audio data and 
picture data, the stronger matching judgment result is 
selected as the final matching result. According to this 
method, the load of the matching judgment processing 
35 is high, however, the average processing procedure un- 
til the matching is established can be reduced. 

Further, the following method may be used. That is, 
only the matching judgment on audio data is performed 
for the first time, and only when the matching is not es- 
40 tablished, the matching processing of picture data is 
performed and the processing continues in accordance 
with the result of the matching processing of picture da- 
ta. As compared with the forementioned method, this 
method has a lower processing load because two kinds 
4S of processing are not necessarily performed at the same 
time for the matching judgment, and the procedure until 
the matching is established can be reduced as com- 
pared with the method of using only one of audio data 
or picture data. The embodiment based on this method 
so can be implemented by modifying steps 102, 105 and 
108. Modified step 12 is shown in Fig .12 

Step 201 represents the same step as step 102 of 
Fig. 7. Step 202 represents a step for performing the 
processing of step 102 by using video data. The modi- 
55 f icatton of steps 1 05 and 1 08 are performed in the same 
manner as step 102 in Fig. 12. 

Priority between matching judgement based on au- 
dio signal and that based on video signal may be 
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changed by replacing step 201 and step 202 in Fig. 12 
each other. 

In the embodiment of Fig. 7, the program ID infor- 
mation 81 is notified at the execution time of step 103, 
which is the time when the reproduced program varies, 
and the in-program time information 83 is issued every 
time when step 103 or 106 is performed. As another 
method, the in-program time information 83 is issued 
only once when the synchronization is established, and 
subsequently no in-program time information 83 is is- 
sued insofar as the synchronization is kept. In this case, 
the external program (the information reproducing pro- 
gram 156 of Fig. 6) calculates the in-program time in 
consideration of the time lapse. In this case, the frequen- 
cy of the issuance of in-program time information by in- 
formation generating program 155 to information repro- 
ducing program 156 through link 163 of Fig. 3 can be 
reduced. In this case, by notifying the out-of-synchro- 
nism through link 163 when the synchronism is judged 
to be lost in steps 107 and 109 (the process goes to the 
direction of NO), the supply of the information by the 
information reproducing program 1 56 can be prevented 
from going ahead arbitrarily in a state where the supply 
of the information is timely deviated from the reproduc- 
ing operation of the program storage device 11 of Fig. 1 . 

Figs. 13 and 14 show examples of display frames 
which are serviced according to the present invention. 

Fig. 1 3 shows an output frame of PC which corre- 
sponds to the output device 20 of Fig. 1. In Fig. 13, ref- 
erence numeral 301 represents a video window which 
represents an output from program storage device 11. 
Reference numeral 302 represents a window which is 
generated by program-associated information repro- 
ducing device 15. 

In this example, the program storage device 11 re- 
produces a news program. A list of key words which are 
associated with the news program is displayed on win- 
dow 302. Keywords such as technical words, contem- 
porary words, etc. used in the news program are added 
newly in the list and the list scrolls in synchronism with 
the progress of the news program displayed on the vid- 
eo window 301 . 

In this embodiment, the program-associated infor- 
mation data stored in storage device 1 7 are assumed to 
have a hypertext structure. Fig. 14 shows a display 
frame to which the display frame of Fig. 13 is shifted 
when any item of the keywords list on the window 302 
is specified through an input device such as a remote 
controller or the like by a user. 

Reference numeral 303 represents a window for 
showing the detailed contents of the specified item, and 
texts or figures are used to show the detailed contents 
in this embodiment. 

According to the information reproducing apparatus 
of the present invention, a TV program broadcasted by 
a broadcast station or the like and program-associated 
information which is generated in association with the 
program by the broadcast station itself or a third party 



other than the broadcast station can be received by 
viewership while the program-associated information 
stored in PC or the like is reproduced in synchronism 
with the reproduction of the TV program recorded in 

s VCR or the like. 

Further, according to the output system of the pro- 
gram ID information and the in-program time information 
of the present invention, when a program which is tem- 
porarily stored in VCR or the like is reproduced at an 

10 arbitrary timing or when special reproduction is per- 
formed on the program, information as to which program 
is currently reproduced and which portion (time) in the 
program is currently reproduced can be generated by 
using information such as audio data or the like which 

is are output from the VCR. Such information is used to 
reproduce the program-associated information in syn- 
chronism with the reproduction of the TV program. 



1. An information reproducing apparatus comprising: 

a program storage device for storing a program; 
2$ a program information outputting device for 

generating and outputting program information 
on the basis of information which is recorded 
together with said program in said program 
storage device; and 
30 a program-associated information reproducing 

apparatus for selecting suitable program-asso- 
ciated information according to said program 
information, and reproducing a suitable portion 
of said program-associated information ac- 
3S cording to said program information. 

2. The information reproducing apparatus as claimed 
in claim 1 , wherein said program information com- 
prises program ID information and in-program time 

40 information. 

3. An information reproducing apparatus comprising: 

a program storage device for storing a program; 

45 a program information outputting device for 

generating program information; and 
a program-associated information reproducing 
apparatus for selecting suitable program-asso- 
ciated information according to said program 

50 information and reproducing a suitable portion 

of said program-associated information ac- 
cording to said program information, 
wherein when said program storage device 
records said program, auxiliary information 
which is generated by said program information 
outputting device is recorded together with said 
program, and when said program storage de- 
vice reproduces said program, said program 
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storage device reproduces said auxiliary infor- 
mation together with said program and said 
program information outputting device calcu- 
lates said program information on the basis of 
said auxiliary information reproduced. 

4. The information reproducing apparatus as claimed 
in claim 3, wherein said program information com- 
prises program ID information and in-program time 
information. 

5. An information reproducing apparatus comprising: 

a program storage device for storing a program; 
a program information outputting device for 
generating program information; and 
a program-associated information reproducing 
apparatus for selecting suitable program-asso- 
ciated information according to said program 
information and reproducing a suitable portion 
of said program-associated information ac- 
cording to said program information, 
wherein when said program storage device 
records said program, on the basis of the con- 
tents of said program, said program information 
outputting device generates and stores auxilia- 
ry information which is subjected to comparison 
judgment by means of a pattern matching or the 
like when said program storage device repro- 
duces said program, and when said program 
storage device reproduces said program, said 
program information outputting device per- 
forms the comparison judgment between the 
contents of said program received from said 
program storage device and said auxiliary in- 
formation by the pattern matching or the like to 
calculate said program information. 

6. The information reproducing apparatus as claimed 
in claim 5, wherein said program information com- 
prises program ID information and in-program time 
information. 

7. An information reproducing apparatus comprising: 

a program storage device for storing a program; 
a program information outputting device for 
generating program information; and 
a program-associated information reproducing 
apparatus for selecting suitable program-asso- 
ciated information according to said program 
information and reproducing a suitable portion 
of said program-associated information ac- 
cording to said program information, 
wherein when said program storage device re- 
produces said program, said program informa- 
tion outputting device performs a comparison 
judgment between the contents of said pro- 



gram received from said program storage de- 
vice and auxiliary information which is prepared 
at the external and is stored, by means of pat- 
tern matching or the like. 

5 

8. The information reproducing apparatus as claimed 
in claim 7, wherein said program information com- 
prises program ID information and said in-program 
time information. 

10 

9. An information reproducing apparatus comprising: 

a program storage device for storing a program; 
a program information outputting device for 
15 generating program information; and 

a program-associated information reproducing 
apparatus for selecting suitable program-asso- 
ciated information according to said program 
information and reproducing a suitable portion 
20 of said program-associated information ac- 

cording to said program information, 
wherein when said program storage device re- 
produces said program, said program informa- 
tion outputting device performs a comparison 
25 judgment between the contents of said pro- 

gram received from said program storage de- 
vice and auxiliary information which is obtained 
from the external, by means of pattern match- 
ing or the like. 

30 

10. The information reproducing apparatus as claimed 
in claim 9, wherein said program information com- 
prises program ID information and in-program time 
information. 

3S 

11. A program information generating method which 
comprises: 

a step of generating and storing audio signal 
40 patterns for one program or more programs, 

a step of selecting one of said audio signal pat- 
terns which corresponds to a currently repro- 
duced audio signal pattern by means of pattern 
matching, and 

45 a step of generating a program information 

which is associated to said audio signal pattern 
selected. 

12. A program information generating method which 
so comprises: 

a step of generating and storing video signal 
patterns for one program or more programs, 
a step of selecting one of said video signal pat- 
55 terns which corresponds to a currently repro- 

duced video signal by means of pattern match- 
ing, and 

a step of generating a program information 
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which is associated to said video signal pattern 
selected. 

13. A program information generating method which 
comprises: s 

a step of generating and storing audio signal 
patterns and video signal patterns for one pro- 
gram or more programs, 

a step of selecting one of said audio signal pat- io 
terns which corresponds to a currently repro- 
duced audio signal pattern by means of pattern 
matching, 

a step of selecting one of said video signal pat- 
terns which corresponds to a currently repro- is 
duced video signal pattern by means of pattern 
matching, 

a step of determining a portion in said one pro- 
gram or more programs by selecting alterna- 
tively said selected audio signal pattern or said 20 
selected video signal pattern based on intensity 
of pattern matching, 

a step of generating a program information 
which is associated to said determined portion. 

25 

14. A program information generating method which 
comprises: 

a first step of generating and storing audio sig- 
nal patterns and video signal patterns for one 30 
program or more programs, 
a second step of selecting one of said audio sig- 
nal patterns which corresponds to a currently 
reproduced audio signal pattern by means of 
pattern matching, 35 
a third step of selecting one of said video signal 
patterns which corresponds to a currently re- 
produced video signal pattern by means of pat- 
tern matching, 

a fourth step of determining a portion in said 40 
one program or more programs by preferential- 
ly selecting the result of said third step and sec- 
ondly selecting the result of said fourth step, 
and 

a fifth step of generating a program information 45 
which is associated to said determined portion, 
wherein third step may be omitted when the re- 
sult of said second step is definite. 

15. A program information generating method which so 
comprises: 

a first step of generating and storing audio sig- 
nal patterns and video signal patterns for one 
program or more programs, ss 
a second step of selecting one of said video sig- 
nal patterns which corresponds to a currently 
reproduced video signal pattern by means of 



pattern matching, 

a third step of selecting one of said audio signal 
patterns which corresponds to a currently re- 
produced audio signal pattern by means of pat- 
tern matching, 

a fourth step of determining a portion in said 
one program or more programs by preferential- 
ly selecting the result of said third step and sec- 
ondly selecting the result of said fourth step, 
and 

a fifth step of generating a program information 
which is associated to said determined portion, 
wherein third step may be omitted when the re- 
sult of said second step is definite. 
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